Skip to content

Validate test binaries upon code changes#780

Open
elizabethteng wants to merge 101 commits intov2.3from
et-validation
Open

Validate test binaries upon code changes#780
elizabethteng wants to merge 101 commits intov2.3from
et-validation

Conversation

@elizabethteng
Copy link
Contributor

@elizabethteng elizabethteng commented Dec 18, 2025 •
edited
Loading

  • enhance 1Z_sun_binaries_suite.py to save evolution data to h5 file in ./outputs
  • update evolve_binaries.sh to reflect output saving
  • write validate_binaries.sh (script for single-line command for comparing branches)
    • call evolve_binaries.sh for new branch
    • call compare_runs.py and save output to file in ./outputs for difference report
    • give option to compare two branches and evolve both, or compare new branch to passed h5 file
  • extract data from runs and save to file
  • write compare_runs.py (compares two h5 files and reports differences)
    • outline
    • quantitative differences beyond thresholds
    • qualitative differences to evolution
    • differences to warnings and errors
  • generate baseline.h5
  • extend to other metallicities
  • tweaks suggested during Mar 12 meeting (They are in your physical notebook!!)

@elizabethteng elizabethteng changed the title Create validate_binaries.sh Validate test binaries upon code changes Dec 18, 2025
elizabethteng and others added 10 commits December 18, 2025 09:18
@elizabethteng
Update 1Zsun_binaries_suite.py to fix args in function calls
91c3d5f
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
8239290 
for more information, see https://pre-commit.ci
@elizabethteng
Update authors
1a01213
@elizabethteng
remove alarm calls that were never set
e967e01 
Removed signal alarm handling in exception block.
@elizabethteng
Enhance evolve_binaries.sh for better logging and error handling
e5c07ce 
Updated the script to log output and check for errors.
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
1a82ac8 
for more information, see https://pre-commit.ci
@elizabethteng
Fix argument passing for evolve_binaries function
5525ea8
@elizabethteng
fill in all code for validate_binaries.sh
138d804 
Updated script to accept branch and suffix as arguments, streamlined output file naming.
@elizabethteng
starting point of compare_runs.py for comparing two h5 files
12a528e
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
9140a20 
for more information, see https://pre-commit.ci
maxbriel
maxbriel reviewed Dec 20, 2025
View reviewed changes
@elizabethteng elizabethteng marked this pull request as ready for review March 9, 2026 23:20
@sgossage sgossage mentioned this pull request Mar 11, 2026
Open
6 tasks
elizabethteng and others added 12 commits March 11, 2026 15:39
@elizabethteng
Update validate_binaries.sh to add guidance about tolerances
fe10943
@elizabethteng
Update README.md to add guidance about tolerances
a4fa52c
@elizabethteng
Update README.md
e6ae3c2
@elizabethteng
Update binaries_suite.py to save path_to_posydon_data in the H5 metadata
28e6af9
@elizabethteng
Update generate_baseline.sh to save PATH_TO_POSYDON_DATA to baseline_…
0bab574 
…info.txt
@elizabethteng
Update binaries_suite.py to check how many binaries ran and save to m…
b1f5166 
…etadata in h5
@elizabethteng
Update generate_baseline.sh to check if any binaries didn't run
f2e9361
@elizabethteng
Update compare_runs.py to eliminate redundancies in report (missing b…
5c505c8 
…inary vs failed binary)
@elizabethteng
Merge remote-tracking branch 'origin/v2.3' into et-validation
b47dd21
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
64be558 
for more information, see https://pre-commit.ci
@elizabethteng
added more descriptive errors to evolve_binaries
d0555a4
@elizabethteng
Merge branch 'et-validation' of https://github.com/POSYDON-code/POSYDON…
7d3290f 
… into et-validation
@sgossage
Copy link
Contributor

sgossage commented Mar 19, 2026 •
edited
Loading

@elizabethteng , during a dev meeting we saw that there were some differences between values when comparing two branches according to these tests. I'm planning to help by looking into the source of those differences. Can you please post here which two branches produced those differences if you have them on hand?

@sgossage sgossage requested review from a team March 19, 2026 19:43
@sgossage sgossage added enhancement New feature or request labels Mar 19, 2026
@sgossage sgossage modified the milestone: v2.3 Mar 19, 2026
@sgossage sgossage requested review from sgossage March 19, 2026 20:54
@sgossage
Copy link
Contributor

sgossage commented Mar 24, 2026

At least in my testing, I noticed differences between branches that would occur only for S1_lg_mdot during step_detached.

This difference indeed seems to be caused by the randomization used in bondi_hoyle during this step. I was able to resolve the differences using PR#826. Creating a base line with the branch sg_v2.3_rng_for_steps and fixing the RNG seed, then comparing to the same branch with the same and different seeds produced: 0 differences when using the same seed and similar differences that I saw when comparing e.g. main and v2.3 when using seeds different from the base line.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@maxbriel maxbriel maxbriel left review comments

@sgossage sgossage Awaiting requested review from sgossage

At least 1 approving review is required to merge this pull request.

Assignees

@elizabethteng elizabethteng

Labels

enhancement New feature or request

Projects

None yet

Milestone

v2.3

Development

Successfully merging this pull request may close these issues.

3 participants

@elizabethteng @sgossage @maxbriel